In order to effectively protect video information, according to the characteristics of H.265/HEVC (High Efficiency Video Coding), a scheme combining transform coefficient scrambling and syntax element encryption was proposed. For Transform Unit (TU), the TU with the size of 4×4 was scrambled by Arnold transform. At the same time, a shift cipher was designed, and the cipher was initialized according to the approximate distribution rule of the Direct Current (DC) coefficient of the TU, and the DC coefficients of TU with the size of 8×8, 16×16 and 32×32 were shifting encrypted using encryption map generated by Arnold transform. For some of the syntax elements with bypass coding used in the entropy coding process, the Logistic chaotic sequence was used for encryption. After encryption, the Peak Signal-to-Noise Ratio (PSNR) and Structual Similarity (SSIM) of the video were decreased by 26.1 dB and 0.51 respectively on average, while the compression ratio was only decreased by 1.126% and the coding time was only increased by 0.17%. Experimental results show that under the premise of ensuring better encryption effect and less impact on bit rate, the proposed scheme has less extra coding overhead and is suitable for real-time video applications.